eval(retrieval): add content_mrr@10 passage-rank metric#248
Conversation
The golden retrieval eval measured document-level rank (mrr@10, over expectedDocumentSubstrings) and content-level *recall* (contentRecall@5, binary "is the answer passage in the top 5"), but nothing measured how HIGH the answer-bearing passage ranks. That left rerank/chunking changes that lift the right passage from #5 to #1 unprovable: content recall stays 1.0 across that move and doc-level mrr is blind to passage order within a document. Add contentReciprocalRankAt10 — the mean over a case's expected content terms of the reciprocal rank of the earliest top-10 result carrying that term (1.0 iff every term rides the rank-1 passage). Surface it in the summary as content_mrr_at_10, averaged only over cases that declare content terms (content_mrr_case_count) so structural zeros don't dilute the signal, plus a per-case contentRR@10 column and the human summary line. Purely additive: the summary type is inferred, so reindex-eval-gate and the other summary consumers are unaffected. Guarded by a focused test proving the metric drops to 0.5 when the answer passage is demoted below a distractor while content recall stays 1.0. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fcc931ad11
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The retrievalLimitForGoldenCase fix landed without a prettier pass, failing the required verify check (format:check). Formatting-only; no logic change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b4f273427b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Two remaining P2 review findings on the passage-rank metric: - Quality-eval path truncation: runRetrievalQualityCases (eval-quality.ts) still fetched topK=testCase.topK, so content_mrr@10 silently degraded to content_mrr@topK for the many fixtures with topK<10. Now uses retrievalLimitForGoldenCase(=max(topK,10)), matching the standalone runner. hit@K semantics are unchanged (evaluateGoldenRetrievalCase still slices to the case's topK), so only positions 9-10 are exposed for the @10 metric. - Missing first-class table/visual evidence: resultContentEvidenceText matched table_facts and a few image fields but not typed index-unit content or the accessible table markdown / table rows, so a hit carrying the answer only in a medication-chart row / risk-matrix cell / flowchart step was invisible to contentReciprocalRankAt10. Now includes index_unit title+content and image accessibleTableMarkdown + flattened tableRows. The retrieval_synopsis stays in the evidence text: it is the stored first-class passage representation used in ranking/display, and the metadata-distractor test already guards the title/file/section vector that the earlier fix excluded. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Addressed the Codex review threads (commits
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a1a6b7f345
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| query: testCase.query, | ||
| ownerId, | ||
| topK: testCase.topK, | ||
| topK: retrievalLimitForGoldenCase(testCase), |
There was a problem hiding this comment.
When npm run eval:retrieval:latency runs, this path now upgrades every fixture with topK < 10 to topK: 10. searchChunksWithTelemetry uses topK throughout selection/reranking and visual-evidence attachment, so this changes the workload being timed rather than only fetching extra rows for the new @10 quality metric; latency reports for the existing topK=8 cases can regress or stop being comparable even when the production topK=8 path is unchanged. Gate the larger fetch to quality/combined scoring, or explicitly separate latency runs for topK=10.
Useful? React with 👍 / 👎.
| return normalized( | ||
| [ | ||
| result.title, | ||
| result.file_name, | ||
| result.section_heading, | ||
| result.section_path?.join(" "), | ||
| result.retrieval_synopsis, | ||
| result.content, | ||
| tableFactText, | ||
| imageText, | ||
| ] | ||
| .filter(Boolean) | ||
| .join(" "), | ||
| [result.retrieval_synopsis, result.content, tableFactText, imageText, indexUnitText].filter(Boolean).join(" "), |
There was a problem hiding this comment.
For memory-boosted results, downstream answer/source paths use result.memory_cards[*].content as structured evidence, but the new content-rank evidence text omits it. In broad-summary or comparison cases where the top chunk carries an expected term only through an attached memory card, contentReciprocalRankAt10 records a miss even though the answer path has usable evidence, creating false passage-rank regressions. Include memory-card content in this evidence text alongside table, visual, and index-unit content.
Useful? React with 👍 / 👎.
| export function retrievalLimitForGoldenCase(testCase: GoldenRetrievalCase) { | ||
| return Math.max(testCase.topK, 10); |
There was a problem hiding this comment.
Captured RAG eval cases are converted with expectedContentTerms: [] and topK: 8, but this helper still upgrades them to a 10-result retrieval even though they are excluded from content_mrr@10. Because searchChunksWithTelemetry uses topK to size candidate budgets and selection before the evaluator later slices back to hit@8, these cases can pass or fail based on candidates/ranking that the actual topK=8 path would not have produced; return the original topK when there are no content terms to score.
Useful? React with 👍 / 👎.
| ] | ||
| .filter(Boolean) | ||
| .join(" "), | ||
| [result.retrieval_synopsis, result.content, tableFactText, imageText, indexUnitText].filter(Boolean).join(" "), |
There was a problem hiding this comment.
When a table hit carries the answer only in table_facts[*].metadata fields such as accessible_table_markdown, table_text_snippet, or cells, this evidence text still reduces table facts to the parsed scalar columns before scoring contentReciprocalRankAt10. The answer and verification paths render/check those metadata snippets, so table-heavy cases can be scored as contentRR misses even though the ranked result contains usable table evidence; add the table-fact metadata text to the evidence string.
Useful? React with 👍 / 👎.
1 participant
What & why
The golden retrieval eval measured document-level rank (
mrr@10, overexpectedDocumentSubstrings) and content-level recall (content_recall@5, binary "is the answer passage in the top 5") — but nothing measured how high the answer-bearing passage ranks. That left rerank/chunking changes (plan items RC8/RC10/RC13) unprovable: a change that lifts the right passage from #5→#1 leaves content recall at 1.0 and doc-levelmrr@10is blind to passage order within a document.This adds
contentReciprocalRankAt10— the mean over a case's expected content terms of the reciprocal rank of the earliest top-10 result carrying that term (1.0 iff every term rides the rank-1 passage). Surfaced ascontent_mrr_at_10, averaged only over cases that declare content terms (content_mrr_case_count) so structural zeros don't dilute the signal, plus a per-casecontentRR@10column and a human-summary line.This is the Phase-2 enabler from the RAG plan ("build the missing chunk-level metric first so ranking changes are provable"). Gate future ranking/chunking changes on it.
Live baseline (golden set, project
sjrfecxgysukkwxsowpy)The new metric already discriminates where the existing ones are blind:
lithium-therapy-monitoring:rr@10=0.00(doc-name unmatched in top-10) butcontentRR@10=0.63— answer content is ranking; the doc miss is a naming artifact, not a content miss.opioid-withdrawal-doses:rr@10=0.25butcontentRR@10=1.00— answer passages at rank 1.insomnia-assessment-management,schizophrenia-overviewat 0.50) become visible rerank-tuning targets.Safety / scope
ReturnType<…>), soreindex-eval-gate.tsand the other summary consumers are unaffected.contentExpectationAlternatives/textContainsClinicalTerm/resultContentText).Tests / checks
tests/eval-retrieval.test.ts+tests/eval-quality.test.ts(21/21) · typecheck · eslint · prettiereval:retrieval:quality— 24/24 pass, baseline aboveNote: distinct from #243's "answer-targeting metric", which is a separate answer-quality metric in
eval-answer-quality.ts— no overlap.